Problems at the intersection of vision and language are of significantimportance both as challenging research questions and for the rich set ofapplications they enable. However, inherent structure in our world and bias inour language tend to be a simpler signal for learning than visual modalities,resulting in models that ignore visual information, leading to an inflatedsense of their capability. We propose to counter these language priors for the task of Visual QuestionAnswering (VQA) and make vision (the V in VQA) matter! Specifically, we balancethe popular VQA dataset by collecting complementary images such that everyquestion in our balanced dataset is associated with not just a single image,but rather a pair of similar images that result in two different answers to thequestion. Our dataset is by construction more balanced than the original VQAdataset and has approximately twice the number of image-question pairs. Ourcomplete balanced dataset is publicly available at www.visualqa.org as part ofthe 2nd iteration of the Visual Question Answering Dataset and Challenge (VQAv2.0). We further benchmark a number of state-of-art VQA models on our balanceddataset. All models perform significantly worse on our balanced dataset,suggesting that these models have indeed learned to exploit language priors.This finding provides the first concrete empirical evidence for what seems tobe a qualitative sense among practitioners. Finally, our data collection protocol for identifying complementary imagesenables us to develop a novel interpretable model, which in addition toproviding an answer to the given (image, question) pair, also provides acounter-example based explanation. Specifically, it identifies an image that issimilar to the original image, but it believes has a different answer to thesame question. This can help in building trust for machines among their users.
展开▼
